A) Analyzing the nodes.csv file (metadata):

B) Examining metadata files that come with dataset:


--- Project Digest Readme ----

The files in this directory contain sensor data and the associated meta-data that will enable parsing the sensor values.

Overview

This sensor data digest contains the following files:

These files will be described in-depth in the following sections.

Sensor Data

The sensor data file is an aggregate of all published data from the project's nodes. By published, we mean:

The data.csv.gz file is a compressed CSV with the following, but not limited to, columns:

These fields will always be provided as a header, for example:

timestamp,node_id,subsystem,sensor,parameter,value_raw,value_hrf
2017/09/09 22:12:44,001e0610ba8f,lightsense,hih4030,humidity,NA,32.18
2017/09/09 22:12:44,001e0610ba8f,lightsense,hih4030,temperature,NA,48.55
2017/09/09 22:12:44,001e0610ba8f,lightsense,ml8511,intensity,9643,NA
2017/09/09 22:12:44,001e0610ba8f,lightsense,tmp421,temperature,NA,43.81
2017/09/09 22:12:44,001e0610ba8f,metsense,hih4030,humidity,450,NA
2017/09/09 22:12:44,001e0610ba8f,metsense,htu21d,humidity,NA,41.15
2017/09/09 22:12:44,001e0610ba8f,metsense,htu21d,temperature,NA,24.1
2017/09/09 22:12:44,001e0610ba8f,metsense,metsense,id,00001814B7E8,00001814B7E8
2017/09/09 22:12:44,001e0610ba8f,metsense,pr103j2,temperature,839,NA

Sensor data is ordered by ascending timestamp.

Additional information such each node's coordinates or each sensor units can be found in the metadata. More information about these will be provided in the next two sections.

A sensor values may be marked NA, indicating that either the raw or HRF value is unavailable.

Node Metadata

The node metadata provides additional information about each of a project's nodes. This file is a CSV with the following fields:

These fields will always be provided as a header, for example:

node_id,project_id,vsn,address,lat,lon,description,start_timestamp,end_timestamp
001e0610ba46,AoT_Chicago,004,State St & Jackson Blvd Chicago IL,41.878377,-87.627678,AoT Chicago (S) [C],2017/10/09 00:00:00,
001e0610ba3b,AoT_Chicago,006,18th St & Lake Shore Dr Chicago IL,41.858136,-87.616055,AoT Chicago (S),2017/08/08 00:00:00,
001e0610ba8f,AoT_Chicago,00D,Cornell & 47th St Chicago IL,41.810342,-87.590228,AoT Chicago (S),2017/08/08 00:00:00,
001e0610ba16,AoT_Chicago,010,Ohio St & Grand Ave Chicago IL,41.891964,-87.611603,AoT Chicago (S) [C],2017/12/01 00:00:00,2018/06/04 00:00:00

Additional details about a node are contained in the description field. The letters inside the brackets [ ] indicate:

Sensor Metadata

The sensor metadata provides additional information about each of the sensors published by the project. This file is a CSV with the following fields:

These fields will always be provided as a header, for example:

ontology,subsystem,sensor,parameter,hrf_unit,hrf_minval,hrf_maxval,datasheet
/sensing/meteorology/pressure,metsense,bmp180,pressure,hPa,300,1100,"https://github.com/waggle-sensor/sensors/blob/master/sensors/airsense/bmp180.pdf"
/sensing/meteorology/temperature,metsense,bmp180,temperature,C,-40,125,"https://github.com/waggle-sensor/sensors/blob/master/sensors/airsense/bmp180.pdf"
/sensing/meteorology/humidity,metsense,hih4030,humidity,RH,0,100,"https://github.com/waggle-sensor/sensors/blob/master/sensors/airsense/htu4030.pdf"
/sensing/meteorology/humidity,metsense,htu21d,humidity,RH,0,100,"https://github.com/waggle-sensor/sensors/blob/master/sensors/airsense/htu21d.pdf"
/sensing/meteorology/temperature,metsense,htu21d,temperature,C,-40,125,"https://github.com/waggle-sensor/sensors/blob/master/sensors/airsense/htu21d.pdf"

More in-depth information about each sensor can be found at: https://github.com/waggle-sensor/sensors

Provenance Metadata

The provenance metadata provides additional information about the origin of this project digest. This file is a CSV with the following fields:

These fields will always be provide as a header, for example:

data_format_version,project_id,data_start_date,data_end_date,creation_date,url
1,AoT_Chicago.complete,2017/03/31 00:00:00,2018/04/10 15:34:36,2018/04/10 15:34:36,http://www.mcs.anl.gov/research/projects/waggle/downloads/datasets/AoT_Chicago.complete.latest.tar.gz

Disclaimer

Although our goal is to provide stable metadata files, please consider these as in-development. If you do write tools which process them, we strongly recommend taking advantage of the metadata headers and processing the files as CSV when applicable in order to accommodate future changes.

C) Analyzing the sensors.csv file (metadata):

image.png

image.png

image.png

NOTE: 

The Waggle Platform is an open source software and hardware platform
for intelligent sensors with advanced edge computing and support for
machine learning.  The Waggle Platform is used by several wireless
sensor projects, including the Array of Things project
(https://arrayofthings.github.io).  For more information on the Waggle
Platform, see:

See http://www.wa8.gl for details.

For details on the open source software license for Waggle components,
please see the file LICENSE.waggle.txt. Some source code files may not
have been originally authored by members of the Waggle team.  In such
cases, a small note describing the modifications for Waggle have been
added to the original copyright and license.

looking at just the particular sensor 'metsense', which we know contains the following variables:

image.png

i.e. the 'sensor' column is the 'part name' from the spec sheet, we know that HIH4030 is a humidity sensor for instance, there should be 9ish main sensors within that subsystem metsense

HUGE AMOUNT OF DATA: 

data_start_date: 9/14/2016 0:00

data_end_date:  9/12/2021 1:20